Add with_output version AppendAttention #3302
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
yuanlehome
reviewed
Aug 11, 2025
| self.causal, | ||
| self.speculative_method is not None, | ||
| )[0] | ||
| if self.use_output: |
There was a problem hiding this comment.
只修改这个文件还不行，全局搜一下所有调用这个append_attention的地方
fastdeploy/model_executor/layers/attention/ops/append_attention.py
Outdated
Show resolved
Hide resolved
|
麻烦再丰富一下PR描述，说明一下改造的背景、目标 |
lizhenyun01
reviewed
Aug 12, 2025
fastdeploy/model_executor/layers/attention/append_attn_backend.py
Outdated
Show resolved
Hide resolved
fastdeploy/model_executor/layers/attention/append_attn_backend.py
Outdated
Show resolved
Hide resolved
gongshaotian
approved these changes
Aug 21, 2025
YuanRisheng
previously approved these changes
Aug 21, 2025
gongshaotian
previously approved these changes
Aug 26, 2025
…into append_attn_pr
DrRyanHuang
suggested changes
Aug 28, 2025
There was a problem hiding this comment.
除了下面俩直接能改的，还有 append_attention 申明为自定义算子的这部分，Outputs 也要改一下
因为咱 append_attention 不是不需要输出 qkv_out 了嘛，所以删掉它
PD_BUILD_STATIC_OP(append_attention)
.Inputs({"qkv",
......
.Outputs({"fmha_out", "qkv_out", "key_cache_out", "value_cache_out"}) # <--- 这一行
.SetInplaceMap({{"key_cache", "key_cache_out"},
改成
.Outputs({"fmha_out", "key_cache_out", "value_cache_out"})
PS: 自定义算子的注册出问题总是直接抛出这种异常，后续主框架也要添加更多上下文信息
terminate called after throwing an instance of 'std:bad_array_new_length'
what(): std: ： bad_array_new_length
custom_ops/gpu_ops/cpp_extensions.cc
Outdated
| const paddle::Tensor &decoder_tile_ids_per_batch, | ||
| const paddle::Tensor &decoder_num_blocks, | ||
| const paddle::Tensor &set_max_lengths, const paddle::Tensor &max_len_kv, | ||
| paddle::Tensor &res, |
There was a problem hiding this comment.
Suggested change
| paddle::Tensor &res, | |
| paddle::Tensor &fmha_out, |
Comment on lines
1126
to
1141
| .Attrs({"compute_type: std::string", | ||
| "cache_quant_type: std::string", | ||
| "use_neox_rotary_style: bool", | ||
| "rope_3d: bool", | ||
| "max_input_length: int", | ||
| "quant_max_bound: float", | ||
| "quant_min_bound: float", | ||
| "out_linear_in_scale: float", | ||
| "encoder_block_shape_q: int", | ||
| "decoder_block_shape_q: int", | ||
| "max_partition_size: int", | ||
| "encoder_max_partition_size: int", | ||
| "speculate_max_draft_token_num: int", | ||
| "causal: bool", | ||
| "speculate_decoder: bool", | ||
| "rms_norm_eps: float"}) |
There was a problem hiding this comment.
这里把 rms_norm_eps 的顺序往前移动一下
Suggested change
| .Attrs({"compute_type: std::string", | |
| "cache_quant_type: std::string", | |
| "use_neox_rotary_style: bool", | |
| "rope_3d: bool", | |
| "max_input_length: int", | |
| "quant_max_bound: float", | |
| "quant_min_bound: float", | |
| "out_linear_in_scale: float", | |
| "encoder_block_shape_q: int", | |
| "decoder_block_shape_q: int", | |
| "max_partition_size: int", | |
| "encoder_max_partition_size: int", | |
| "speculate_max_draft_token_num: int", | |
| "causal: bool", | |
| "speculate_decoder: bool", | |
| "rms_norm_eps: float"}) | |
| .Attrs({"rms_norm_eps: float", | |
| "compute_type: std::string", | |
| "cache_quant_type: std::string", | |
| "use_neox_rotary_style: bool", | |
| "rope_3d: bool", | |
| "max_input_length: int", | |
| "quant_max_bound: float", | |
| "quant_min_bound: float", | |
| "out_linear_in_scale: float", | |
| "encoder_block_shape_q: int", | |
| "decoder_block_shape_q: int", | |
| "max_partition_size: int", | |
| "encoder_max_partition_size: int", | |
| "speculate_max_draft_token_num: int", | |
| "causal: bool", | |
| "speculate_decoder: bool", | |
| }) |
DrRyanHuang
approved these changes
Aug 28, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
背景：cudagraph 捕获过程中的张量地址管理
目的：将attention模块的输出前置，便于cudagraph捕获时的张量地址处理